Image processing method, image processing system, and non-transitory computer readable medium

ABSTRACT

An image processing method according to an embodiment includes a specifying step, an inference step, and an integration step. In the specifying step, a first portion including a region corresponding to an anatomical site of a target and a second portion including a region different from the anatomical site are specified in the image. In the inference step, by using a deep learning model, segmentation of the region corresponding to the anatomical site is performed on the first portion and segmentation of the region different from the anatomical site is performed on the second portion, or classification and detection of an image including the region corresponding to the anatomical site is performed on the first portion and classification and detection of an image including the region different from the anatomical site is performed on the second portion. In the integration step, results of the respective processes are integrated for output.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromChinese Patent Application No. 202210630102.6, filed on Jun. 6, 2022;the entire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an image processingmethod, an image processing system, and a non-transitory computerreadable medium.

BACKGROUND

Deep learning is currently an important technical means of performingimage processing such as segmentation, classification, and detection onan image. A complete deep learning framework includes two main parts: atraining process; and an inference process. The training process is aprocess of training a model by using a training data set with a label(true value: GT, also referred to as Ground Truth). The inferenceprocess is a process of inputting unlabeled live data into an alreadytrained model to acquire an actual detection value.

In the inference process, when the trained model is used to infer animage, the inference results may be inaccurate. Such inaccurateinference results are particularly noticeable with respect to certainparts of the image, for example, edges of an organ, edges of a tumor,and boundary parts such as organ defect sites due to partial organresection.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of the configurationof an image processing system according to the present embodiment;

FIG. 2 is a comparative diagram for explaining features of a processperformed by the image processing system according to the presentembodiment;

FIG. 3 is a flowchart illustrating an example of a process performed bythe image processing system according to the present embodiment;

FIG. 4 is a schematic diagram illustrating preprocessing of a processingprocedure for specifying a first portion and a second portion by animage processing system according to a first embodiment;

FIG. 5A is a schematic diagram illustrating a processing procedure forspecifying a first portion and a second portion by the image processingsystem according to the first embodiment;

FIG. 5B is a schematic diagram illustrating a processing procedure forspecifying a first portion and a second portion by the image processingsystem according to the first embodiment;

FIG. 6A is a schematic diagram illustrating the principle of specifyinga first portion and a second portion by the image processing systemaccording to the first embodiment and the effects thereof;

FIG. 6B is a schematic diagram illustrating the principle of specifyinga first portion and a second portion by the image processing systemaccording to the first embodiment and the effects thereof;

FIG. 6C is a schematic diagram illustrating the principle of specifyinga first portion and a second portion by the image processing systemaccording to the first embodiment and the effects thereof;

FIG. 7 is a schematic diagram illustrating a processing status of stepS200 by the image processing system according to the first embodiment;

FIG. 8 is a schematic diagram illustrating a processing status of stepS300 by the image processing system according to the first embodiment;

FIG. 9A is a schematic diagram illustrating a processing status of stepS500 by the image processing system according to the first embodiment;

FIG. 9B is a schematic diagram illustrating the processing status ofstep S500 by the image processing system according to the firstembodiment;

FIG. 10 is a schematic diagram illustrating a processing procedure forspecifying a first portion and a second portion by an image processingsystem according to a second embodiment;

FIG. 11 is a schematic diagram illustrating a processing status at stepS100 of a deep learning model of the image processing system accordingto the second embodiment;

FIG. 12 is a schematic diagram illustrating a processing status of stepS200 by the image processing system according to the second embodiment;

FIG. 13 is a schematic diagram illustrating a processing status of stepS300 by the image processing system according to the second embodiment;and

FIG. 14 is a schematic diagram illustrating a processing status of stepS500 by the image processing system according to the second embodiment.

DETAILED DESCRIPTION

An image processing method according to an embodiment is an imageprocessing method of performing an inference process on image data byusing a trained deep learning model, and includes a receiving step, asetting step, a specifying step, an inference step, and an integrationstep. The receiving step receives an image. The setting step sets ananatomical site of a target. The specifying step specifies a firstportion and a second portion in the image, the first portion including aregion corresponding to the anatomical site of the target, the secondportion including a region different from the anatomical site of thetarget. By using the deep learning model, the inference step performs,as a first inference process, a segmentation process of specifying theregion corresponding to the anatomical site of the target with respectto the first portion and performs, as a second inference process, asegmentation process of specifying the region different from theanatomical site of the target with respect to the second portion, orperforms, as the first inference process, a classification process ofclassifying and detecting an image including the region corresponding tothe anatomical site of the target with respect to the first portion andperforms, as the second inference process, a classification process ofclassifying and detecting an image including the region different fromthe anatomical site of the target with respect to the second portion.The integration step integrates results of the first inference processand the second inference process and outputs an integrated result.

Embodiments of an image processing system, an image processing method,and a non-transitory computer readable medium according to the presentapplication are described below.

The present embodiment relates to an image processing method, an imageprocessing system, and a non-transitory computer readable medium.Particularly, the present embodiment relates to an image processingmethod and an image processing system that can achieve both accuracy andspeed of an inference process in deep learning in an image processingmethod based on the deep learning.

Deep learning is currently an important technical means of performingimage processing such as segmentation, classification, and detection onan image. A complete deep learning framework includes two main parts: atraining process and an inference process. The training process inputs atraining data set with a label (true value: GT, also referred to asGround Truth) to a model, calculates a target function (loss function)between an output detection result and the true value, and corrects anetwork parameter by a gradient descent method, a stochastic gradientdescent method, or the like to minimize the loss function. The trainingprocess is repeated until an error between the network output detectionresult and the true value satisfies predetermined accuracy, therebybringing the model to a convergence state and reducing an error in aprediction value of the model to complete the training of the model. Theinference process is also a process of inputting unlabeled live datainto an already trained model to acquire an actual detection value.

In the inference process, when the trained model is used to infer animage, inference results may be inaccurate. Such inaccurate inferenceresults are particularly noticeable with respect to certain parts of theimage, for example, edges of an organ, edges of a tumor, and boundaryparts such as organ defect sites due to partial organ resection.

Several methods have been proposed in the related art in order toimprove the accuracy of inference in deep learning. For example, PatentLiterature 1 discloses an artificial intelligence-based medical imagesegmentation method and system that can adaptively employ appropriatealgorithms to perform an inference process on the same set of images.Specifically, the technique of Patent Literature 1 performs segmentationby automatically selecting one or a plurality of appropriate deeplearning models according to features such as image scanning protocolsand organ features of the image. When a plurality of models areselected, an inference result of a previous model and an original imageare concatenated by a method of model concatenation and used as inputfor a next model.

Patent Literature 2 discloses a method and system for automatic machinelearning of organ segmentation that focuses on organ segmentation,performs multi-channel inference by using image patches having differentsizes, and concatenates results of the multi-channel inference as inputfor the last layer of a deep learning network.

Patent Literature 3 discloses a learning device and an inference deviceusing a machine learning model that reduces an average time for dataprocessing by determining whether target image data has a predictableregion that can easily predict an inference result of an inferenceprocess, performing predetermined data processing on a data regiondesignated as a predictable region, and outputting data necessary forinferring a non-predictable region to the machine learning model.

In the related art, in order to improve the accuracy of an inferenceprocess, an attempt has also been made to perform a precise process byusing a modified inference algorithm, and such a modified inferenceprocess is referred to as “enhanced inference”. Specific examples of theenhanced inference include increasing an overlap ratio between imagepatches used during inference when segmenting an image, performinginference on all images when classifying the images, and performinginference again after performing flipping, scaling, or the like on animage when detecting the image.

In the related art, when performing the enhanced inference describedabove, these enhanced inference processes are employed for all images tobe processed, which increases the processing time of an algorithm.Therefore, the related art has a technical problem that it is notpossible to achieve both accuracy and processing speed in imageprocessing using the enhanced inference.

The present embodiment has been made to solve the above problems of therelated art. The present embodiment proposes a new method of achievingboth accuracy and speed of an inference process in a deep learningalgorithm. The present embodiment first divides an image to be processedinto a first portion and a second portion. The first portion is aportion of the image to be processed where the deep learning algorithmis likely to obtain an inaccurate result, for example, a portion near alocation in the image where an inference process result suddenlychanges. The second portion is a portion, other than the first portion,in the image to be processed. The present embodiment performs a firstinference process and a second inference process, which are differentinference processes, on the first portion and the second portion,respectively. For example, the present embodiment performs preciseenhanced inference on the first portion and faster normal inference onthe second portion. Then, the present embodiment combines processingresults of the enhanced inference and the normal inference as aprocessing result for the entire image.

Specifically, an aspect of the present embodiment is an image processingmethod of performing an inference process on image data by using atrained deep learning model, and provides an image processing methodincluding a receiving step of receiving an image, a specifying step ofspecifying a first portion and a second portion in the received image,an inference step of using the deep learning model to perform a firstinference process on the specified first portion and a second inferenceprocess, which is an inference process different from the firstinference process, on the specified second portion, and an integrationstep of integrating results of the first inference process and thesecond inference process and outputting an integrated result.

Another aspect of the present embodiment is an image processing systemthat performs an inference process on image data by using a trained deeplearning model, and provides an image processing system including areceiving device that receives an image, a specifying device thatspecifies a first portion and a second portion in the received image, aninference device that uses the deep learning model to perform a firstinference process on the specified first portion and a second inferenceprocess, which is an inference process different from the firstinference process, on the specified second portion, and an integrationdevice that integrates results of the first inference process and thesecond inference process and outputs an integrated result.

According to the present embodiment, an image to be processed is dividedinto the first portion, which is a portion near a location in the imagewhere an inference process result suddenly changes, and the secondportion other than the first portion. Different inference processes areperformed on the first portion and the second portion, respectively.Specifically, precise enhanced inference is performed on the firstportion and faster normal inference is performed on the second portion.Then, processing results of the enhanced inference and the normalinference are integrated to obtain a processing result for the entireimage. This makes it possible to implement a balance between accuracyand speed during deep learning model inference, and achieve moredesirable inference results.

Embodiments of an image processing system, an image processing method,and a non-transitory computer readable medium according to the presentapplication are described in detail below with reference to theaccompanying drawings. The image processing system, the image processingmethod, and the non-transitory computer readable medium according to thepresent application are not limited by the following embodiments. In thefollowing description, the same components are given common referencenumerals, and redundant description is omitted.

An overview of an image processing system according to the presentembodiment is first described. The image processing system of thepresent application may exist in the form of a medical image diagnosticapparatus such as an ultrasonic diagnostic apparatus, a computedtomography (CT) imaging apparatus, or a magnetic resonance imaging (MRI)imaging apparatus, or exist independently in the form of a workstationor the like.

FIG. 1 is a block diagram illustrating an example of the configurationof the image processing system according to the present embodiment.

An image processing system 1 according to the present embodimentperforms an inference process on an input image by using a deep learningneural network. As illustrated in FIG. 1 , the image processing system 1mainly includes a receiving device 10, a specifying device 20, aninference device 30, and an integration device 40. The receiving device10 receives an image to be processed. The specifying device 20 sets ananatomical site of a target, and specifies a first portion and a secondportion, other than the first portion, in the image received by thereceiving device 10, the first portion including a region correspondingto the anatomical site of the target, the second portion including aregion different from the anatomical site of the target. In the presentembodiment, the first portion is a portion of the image to be processedwhere an algorithm of a deep learning model is likely to obtain aninaccurate result, for example, a portion near a location in the imagewhere an inference process result suddenly changes. Details of the firstportion are described below. The inference device 30 uses a trained deeplearning model to perform a first inference process and a secondinference process, which are different inference processes, on the firstportion and the second portion specified by the specifying device 20,respectively.

The integration device 40 integrates inference results acquired by thedifferent inference processes and outputs an integrated result.

The image processing system 1 may be provided in, for example, an imageprocessing system such as an ultrasonic diagnostic apparatus. In thiscase, the image processing system 1 further includes a control unit, anultrasound probe, a display, an input/output interface, a device body,and the like, which are not illustrated in the drawing. The receivingdevice 10, the specifying device 20, the inference device 30, and theintegration device 40 are provided in the control unit and arecommunicably connected to these ultrasound probe, display, input/outputinterface, device body, and the like. Since the configurations,functions, and the like of the control unit, the ultrasound probe, thedisplay, the input/output interface, and the device body are well knownto those skilled in the art, detailed description thereof is omitted.

A process performed by the image processing system 1 according to thepresent embodiment is described in detail below.

FIG. 2 is a comparative diagram for explaining features of a processperformed by the image processing system according to the presentembodiment.

FIG. 2 illustrates the features of the process according to the presentembodiment by means of comparison with the flow of the related art. InFIG. 2 , the left side denotes a flowchart of image processing in therelated art, and the right side denotes a flowchart of the processperformed by the image processing system according to the presentembodiment. As illustrated in FIG. 2 , in the process performed by theimage processing system according to the present embodiment, step S100,step S200, and step S300 are characteristic steps of the presentembodiment. While the related art performs enhanced inference on anentire image at step S20′, the present embodiment first specifies afirst portion and a second portion in an image to be processed at stepS100. Subsequently, at step S200, the present embodiment performs afirst inference process and a second inference process, which aredifferent inference processes, on the specified first portion and secondportion, respectively. Specifically, as will be described below withreference to FIG. 3 , precise enhanced inference is used for the firstportion, and faster normal inference is used for the second portion.Subsequently, at step S300, the present embodiment integrates processingresults of the enhanced inference and the normal inference to obtain aprocessing result for the entire image. The image processing systemaccording to the present embodiment divides the image to be processedinto the first portion and the second portion and performs the firstinference process and the second inference process, which are differentinference processes, respectively, so that the present embodiment canimplement a balance between accuracy and speed during deep learningmodel inference and achieve more desirable inference results, unlike therelated art that performs enhanced inference on the entire image to beprocessed.

Details of the process performed by the image processing systemaccording to the present embodiment are described in detail below withreference to FIG. 3 .

FIG. 3 is a flowchart illustrating an example of the process performedby the image processing system according to the present embodiment.

First, at step S10, the image processing system 1 receives medical imagedata to be processed by the receiving device 10.

Since the processes of subsequent step S100, step S200, step S300, andstep S30 have been described above with reference to FIG. 2 ,description thereof is omitted.

In accordance with the image processing system according to the presentembodiment, a determination step S400 and a correction step S500 may befurther provided between step S300 and step S30. At step S400, the imageprocessing system 1 determines whether the integrated algorithm resultis satisfied. This step may be performed by setting a predeterminedthreshold in advance for a technical index indicating the result of theinference process and comparing the index of the integrated result withthe threshold, or may be performed by a user of the image processingsystem 1 who makes an artificial determination.

When the determination result at step S400 satisfies the integratedresult (YES at step S400), the process proceeds to step S30, outputs thealgorithm result, and terminates the process. On the other hand, whenthe determination result of step S400 does not satisfy the integratedalgorithm result (NO at step S400), the process proceeds to step S500 tocorrect the first portion specified at step S100. The processes of stepS200 and step S300 are further performed on the corrected first portionand the second portion other than the first portion until thedetermination result at step S400 is “YES”, and the determination ofstep S400 is performed again on the integrated algorithm result. Theprocess of step S500 may be automatically performed by the correctionfunction of the image processing system 1 according to preset rules, ormay be artificially performed by a user via the user interface of theimage processing system 1.

Step S400 and step S500 are not essential. As illustrated in FIG. 2 ,even a method including only step S10, step S100, step S200, step S300,and step S30 can achieve the purpose of the present embodiment andobtain effects of the present embodiment.

The overview of the process performed by the image processing systemaccording to the present embodiment has been described above. Thepresent embodiment is described in detail below using a segmentationprocess and a classification process as examples.

First Embodiment

The first embodiment is an example in which the image processing systemaccording to the present embodiment is applied to image segmentation.

The first embodiment is described in detail below with reference to FIG.4 , FIGS. 5A and 5B, FIGS. 6A to 6C, FIGS. 7 and 8 , and FIGS. 9A and 9Bby using a segmentation process for the liver as an example.

With reference to FIG. 4 , FIGS. 5A and 5B, and FIGS. 6A to 6C, aprocess of specifying a first portion and a second portion at step S100in the first embodiment is first described.

FIG. 4 is a schematic diagram illustrating preprocessing of a processingprocedure for specifying the first portion and the second portion by theimage processing system according to the first embodiment.

FIG. 4 illustrates an example of performing the segmentation process ona single image to be processed. A large rectangular frame in the drawingrepresents the entire image to be processed, four large blocks in therectangular frame represent image patches to be subjected to thesegmentation process, a small dark block in the image patch represents asingle pixel, and the number in the small block represents the inferenceprobability of a pixel. The rectangular frame and the size and number ofthe blocks are merely schematic representations for convenience ofdescription. The size contrast in the drawing is not to match with theactual size scale, and the actual number may be other than the numberillustrated. This point is the same for all attached drawings.

As illustrated in FIG. 4 , when the image processing system according tothe first embodiment performs the segmentation process to the liver onmedical image data, the specifying device 20 first performs normalinference on the entire single image to be processed by the deeplearning model at step S100. The “normal inference” in the presentembodiment refers to light inference having lower accuracy, lowerprocessing load, and higher speed than enhanced inference. Specifically,the normal inference of the present embodiment is inference performedusing large image patches, for example, as illustrated in FIG. 4 . Thespecifying device 20 acquires a matrix having the inference probabilityof each pixel in the entire image by performing light inference on theentire single image to be processed. The inference probability is anexample of an “inference result index” in the present embodiment, andwhen segmentation of a target region (for example, liver) is performedon an image to be processed, the inference probability is theprobability that a pixel indicated by a segmentation result belongs tothe target region (liver). In FIG. 4 , a dark-colored portion in theimage is the target region (liver), and a light-colored backgroundportion is a region other than the target region (liver).

As illustrated in FIG. 4 , inside the dark-colored liver region, thesegmentation result for each pixel obtained by the light inference isthat the pixel belongs to the liver region, so the inference probabilityof each pixel obtained accordingly is 1.0. In the light-colored region,the segmentation result for each pixel is that the pixel does not belongto the liver region, so the inference probability of each pixel obtainedaccordingly is 0.0. In the vicinity of a boundary between thedark-colored region and the light-colored region, the segmentationresult for each pixel is that the pixel belongs to the liver region witha certain probability between 0.0 and 1.0, so accordingly, asillustrated in FIG. 4 , for example, respective inference probabilitiesof four pixels at the boundary are 0.52, 0.50, 0.50, 0.48, and the like.In this way, in the image to be processed illustrated in FIG. 4 , thereis a location in a region near the boundary between the dark-coloredregion and the light-colored region where the inference probability,which is the result of the inference process in the present embodiment,suddenly changes, for example, from 0.0 to a value greater than 0.0 andsmaller than 1.0, or from 1.0 to a value greater than 0.0 and smallerthan 1.0. In the present embodiment, such a region near a location inthe image to be processed where the inference process result suddenlychanges is specified as the first portion.

In the case of the first embodiment in which segmentation is performedon an image, the first portion and the second portion in the image inthe present embodiment are different regions in one image, andspecifically refer to an important region that is important for thesegmentation and an unimportant region that is not important for thesegmentation, respectively.

A procedure for specifying the first portion and the second portion bythe image processing system according to the first embodiment isdescribed in detail below.

FIGS. 5A and 5B are schematic diagrams illustrating a processingprocedure for specifying the first portion and the second portion by theimage processing system according to the first embodiment.

As illustrated in FIGS. 5A and 5B, after obtaining an inferenceprobability matrix for each pixel included in the entire image to beprocessed as illustrated in FIG. 4 , the specifying device 20 specifiesthe first portion and the second portion on the basis of the inferenceprobability matrix.

First, as illustrated in FIG. 5A, in the present embodiment, thespecifying device 20 sets an important region inference probabilityrange with a lower limit value greater than 0.0 and an upper limit valuesmaller than 1.0 on the basis of the obtained inference probabilitymatrix, specifies, as the first portion, an image region includingpixels within the important region inference probability range in whichan inference probability value is set, and specifies an image regionother than the first portion as the second portion.

In FIG. 5A, as an example, the lower limit value is set to 0.05, theupper limit value is set to 0.95, and the important region inferenceprobability range corresponding to the first portion is set as follows.

0.05≤inference probability≤0.95

That is, according to the first embodiment, the present embodimentspecifies, as an important region, an image region including pixels withan inference probability of 0.05 to 0.95, that is, a dark-coloredconstant-width liver outline region in FIG. 5B, and specifies, as anunimportant region, an image region including pixels satisfying0.0≤inference probability <0.05 or 0.95<inference probability ≤1.0, thatis, a light-colored region in FIG. 5B. In the present embodiment, theimportant region and the unimportant region are examples of the firstportion and the second portion, respectively.

With reference to FIGS. 6A to 6C, the principle of the method ofspecifying the first portion and the second portion at step S100 and theeffects thereof are described below.

FIGS. 6A to 6C are schematic diagrams illustrating the principle ofspecifying the first portion and the second portion by the imageprocessing system according to the first embodiment and the effectsthereof, taking the segmentation of a liver, a tumor, and a defect siteof the liver as examples. FIG. 6A is a schematic diagram illustrating areal image as a processing target. FIG. 6B is a schematic diagram forexplaining the principle of setting the important region inferenceprobability range and the effects thereof. FIG. 6C is a schematicdiagram illustrating an image in which the first portion and the secondportion after segmentation are displayed.

As illustrated in FIG. 6A, a real medical image as a processing targetincludes the liver (hatched area in FIG. 6A), which is a segmentationtarget, and defect sites in the liver due to a tumor or surgicalresection, for example (grid areas in FIG. 6A). As described above, thespecifying device 20 performs segmentation on the liver, the tumor, andthe defect sites, which are segmentation targets in FIG. 6A, by usinglight inference, so that an inference probability distributionillustrated in FIG. 6B is obtained. In FIG. 6B, the horizontal axisdenotes the inference probability of each pixel, and the vertical axisdenotes the number of pixels corresponding to the normalized inferenceprobability. It can be seen from FIG. 6B that in the inferenceprocessing result, the number of pixels with an inference probability ofnear 0 or 1 is the largest, and the number of pixels with an inferenceprobability in a range between a lower limit value of the inferenceprobability that is slightly away from 0 and larger than 0 (for example,0.05) and an upper limit value of the inference probability that isslightly away from 1 and smaller than 1 (for example, 0.95) issignificantly small. Therefore, the present embodiment selects such aninference probability range in which the number of pixels issignificantly small as the important region inference probability range,and thereby specifies an important region (first portion) correspondingto the important region inference probability range and an unimportantregion (second portion) other than the important region.

In the present embodiment, the important region inference probabilityrange is set to 0.05 to 0.95, but this is merely an example and theimportant region inference probability range of the present embodimentis not limited thereto. In the present embodiment, the important regioninference probability range is set so that the difference between thelower limit value and 0 is 0.05 (=0.05-0) and the difference between theupper limit value and 1 is 0.05 (=1-0.95), that is, the differencebetween the lower limit value and 0 and the difference between the upperlimit value and 1 are equal to each other; however, the presentembodiment is not limited thereto and the important region inferenceprobability range may be set so that the difference between the lowerlimit value and 0 and the difference between the upper limit value and 1are different from each other. According to the present embodiment, aninference probability range in which the distribution of the number ofpixels with inference probabilities near 0 or 1 is significantly smallmay be selected as the important region inference probability range. Inthe present embodiment, a normalization process is performed on thenumber of pixels to determine a range in which the distribution of thenumber of pixels is significantly small, from the distribution ofnormalized pixels; however, the present embodiment is not limitedthereto. Since the definition that the distribution of the number ofpixels is significantly small and the specifying method thereof canemploy other methods known to those skilled in the art and are not a keypoint of the present embodiment, detailed description thereof isomitted.

In accordance with the image processing system according to the firstembodiment, by setting the important region inference probability rangeto 0.05 to 0.95, an important region (first portion) such as the oneillustrated in FIG. 6C is specified. As can be seen from FIG. 6C, pixelswith an inference probability range between 0.05 and 0.95 constitute theouter edge of the liver with a certain width and the outline of thetumor or the defect side, and the outer edge of the liver and theoutline of the tumor or the defect side are important targets forsegmentation of the medical image to be processed. By the specifyingstep S100 of the present embodiment, the outer edge of the liver and theoutline region of the tumor or the defect side are specified as animportant region (first portion), and a region other than the importantregion is specified as an unimportant region (second portion).

The above is the description of the process of specifying the firstportion and the second portion at step S100 in the first embodiment.

After the process of specifying the first portion and the second portionis terminated, the present embodiment proceeds to the inference processof step S200.

With reference to FIG. 7 , the inference process of step S200 in thefirst embodiment is described below.

FIG. 7 is a schematic diagram illustrating a processing status of stepS200 by the image processing system according to the first embodiment.

After the first portion (important region) and the second portion(unimportant region) are specified at step S100, the inference device 30performs the first inference process and the second inference process,which are different inferences, on the first portion (important region)and the second portion (unimportant region), respectively, at step S200.Specifically, the inference device 30 performs enhanced inference on thespecified first portion as the first inference process. As the enhancedinference, an image patch can be newly divided around the first portionfor inference as illustrated in FIG. 7 .

In addition, examples of the enhanced inference include a test timeaugmentation method such as flipping an image patch including the firstportion and then performing an inference process again or increasing apatch overlap ratio between image patches when performing an inferenceprocess on the first portion. The present embodiment is not limitedthereto, and other enhanced inference methods for segmentation known tothose skilled in the art may be employed as the enhanced inference. Theinference device 30 performs light inference, which has a lowerprocessing load and higher speed, on the second portion as the secondinference process. The inference device 30 can use the results of thelight inference performed on the second portion when specifying thefirst portion and the second portion at step S100. Using the results ofthe light inference at step S100 as is is preferable because it can savethe processing time, reduce the processing load, and increase theprocessing speed. The present embodiment is not limited thereto, andanother light inference may be newly performed on the second portion asthe second inference process.

After the inference process is terminated, the present embodimentproceeds to the integration process of step S300.

With reference to FIG. 8 , the integration process of step S300 in thefirst embodiment is described below.

FIG. 8 is a schematic diagram illustrating a processing status of stepS300 by the image processing system according to the first embodiment.

As illustrated in FIG. 8 , at step S300, the integration device 40integrates the results of the enhanced inference on the first portionand the light inference on the second portion obtained at step S200, anduses the integrated result as a complete inference result.

Specifically, the integration device 40 may integrate the inferenceresults by directly superimposing the results of the enhanced inferenceon the first portion with the results of the light inference on thesecond portion. At step S200, when it is preferable to use the resultsof the light inference performed on the second portion when specifyingthe first portion and the second portion at step S100 as they are, theintegration device 40 may directly overwrite the processing results ofthe enhanced inference performed on the first portion obtained at stepS200 on the results of the light inference on the second portionobtained at step S100. The results of the enhanced inference on thefirst portion and the results of the light inference on the secondportion may be fused and processed with a certain weight. When the liveris a segmentation target, the result of the integration process at stepS300 is a complete liver segmentation result illustrated in FIG. 8 .

Subsequently, the integration device 40 outputs the processing result ofstep S300 as a final result of the inference process.

In accordance with the image processing system according to the presentembodiment, a determination step S400 and a correction step S500 may befurther provided between step S300 and step S30 as a preferredembodiment as described above. At step S400, the image processing system1 determines whether the integrated algorithm result is satisfied. Whenthe determination result at step S400 is “YES”, the process proceeds tostep S30, outputs the algorithm result, and terminates the process. Onthe other hand, when the determination result of step S400 is “NO”, theprocess proceeds to step S500 to correct the first portion and thesecond portion specified at step S100, and returns to the process ofstep S200 again.

Details of the process of the determination step S400 have beenpreviously described with reference to FIG. 3 . With reference to FIG. 9, the process of the correction step S500 is described below.

FIGS. 9A and 9B are schematic diagrams illustrating a processing statusof step S500 by the image processing system according to the firstembodiment.

As illustrated in FIG. 9A, the process of step S500 can be implementedby adjusting the lower limit value and the upper limit value of theimportant region inference probability range. Such adjustment may beautomatically performed by the correction function of the imageprocessing system 1 according to preset rules, or may be artificiallyperformed by a user. As illustrated in FIG. 9B, the process of step S500may be implemented, for example, by directly correcting, via a userinterface, a previously obtained image including an important region(first portion) and an unimportant region (second portion) asillustrated in FIG. 6C. Such correction may be implemented by a user whoclicks a corresponding region in the image with a mouse and drags anddrops the region, or may be automatically performed by the system.

After step S500 is performed, the image processing system according tothe present embodiment further performs the processes of step S200 andstep S300 on the corrected first portion and the second portion otherthan the first portion until the determination result of step S400 is“YES”, and performs the determination of step S400 again on theintegrated algorithm result.

As described above, step S400 and step S500 are not essential. Eventhough the processes of step S400 and step S500 are not performed, theobject of the present embodiment can be similarly achieved and theeffects of the present embodiment can be similarly obtained.

Summary of First Embodiment

According to the first embodiment, the inference process of the presentembodiment is used to perform segmentation on an image. The firstportion is an important region important for image segmentation, and thesecond portion is an unimportant region not important for the imagesegmentation.

The specifying step performs segmentation on an image by using a deeplearning model, acquires an inference result index indicating asegmentation result for each pixel in the image, specifies, as animportant region, a region including pixels for which the inferenceresult index is between a predetermined lower limit value and apredetermined upper limit value, and specifies a region other than theimportant region as an unimportant region. The inference step performsenhanced segmentation based on enhanced inference on the importantregion, uses a result of the enhanced segmentation as an inferenceresult for the important region, and uses a segmentation result for theunimportant region at the specifying step as is as an inference resultfor the unimportant region. In this way, according to the presentembodiment, an image to be processed is divided into the first portionand the second portion other than the first portion, and the firstinference process and the second inference process, which are differentinference processes, on the first portion and the second portion,respectively. Specifically, precise enhanced inference is performed onthe first portion and faster normal inference is performed on the secondportion. Subsequently, results of the enhanced inference and the normalinference are integrated to obtain a processing result for the entireimage. This makes it possible to implement a balance between accuracyand speed during the deep learning model inference, and achieve moredesirable inference results.

Since details of the deep learning and a method to segment an imageusing the deep learning model are well known in the art and are not keypoints of the present embodiment, detailed description thereof isomitted.

In the above description, inference probability is used as an example ofthe inference result index of the first embodiment; however, the presentembodiment is not limited thereto and in addition to the inferenceprobability, for example, inference uncertainty of the deep learningmodel may be used as the inference result index. Alternatively, an imageprocessing method in the related art may be used to detect a differencein pixels of an image, and the detected difference may be used as theinference result index to specify the first portion of the image. Sincedetails of the inference uncertainty are described in detail, forexample, in Alex Kendall and Yarin Gal's paper “What Uncertainties Do WeNeed in Bayesian Deep Learning for Computer Vision?” (31^(st) Conferenceon Neural Information Processing Systems (NIPS 2017), Long Beach, CA,USA.), detailed description thereof is omitted.

Second Embodiment

The first embodiment described above is an example in which the imageprocessing system according to the present embodiment is applied toimage segmentation; however, the embodiment is not limited thereto. Forexample, the image processing system according to the present embodimentcan also be applied to image classification.

The second embodiment is an example in which the image processing systemaccording to the present embodiment is applied to image classification.

The second embodiment is described in detail below with reference toFIGS. 10 to 14 , taking a classification process for organ localizationof the liver as an example.

In the description of the second embodiment, the differences from thefirst embodiment described above are mainly described, and the sameconfigurations as in the first embodiment described above are denoted bythe same reference numerals and description thereof is omitted.

With reference to FIGS. 10 and 11 , a process of specifying a firstportion and a second portion at step S100 in the second embodiment isfirst described.

In the case of the second embodiment in which a classification processfor organ localization is performed on an image, the first portion andthe second portion in the image in the present embodiment are differentranges of the number of images in a plurality of images. Specifically,the first portion and the second portion refer to the range of thenumber of important images that are important for classification and therange of the number of unimportant images that are not important for theclassification, respectively, among the plurality of images.

FIG. 10 is a schematic diagram illustrating a processing procedure forspecifying a first portion and a second portion by an image processingsystem according to the second embodiment.

FIG. 10 illustrates an example of performing a classification processfor organ location of the liver, for example, on a series of images tobe processed. As illustrated in FIG. 10 , the image processing systemaccording to the second embodiment, for example, classifies and detectsa plurality of images by using a multi-channel deep learning model withrespect to a set of a plurality of consecutive 2D images obtained forthe upper body of a human, specifies, as the range of the number ofimportant images (first portion), the range of a predetermined number ofimages near an image in which the classification and detection resultsuddenly changes among the plurality of images, and specifies the rangeof the number of other images as the range of the number of unimportantimages (second portion). Specifically, the image processing systemaccording to the second embodiment performs classification and detectionfor each group of the images by using a deep learning model withmulti-channel input, and specifies, as the range of the number ofimportant images, a group including images with a sudden change in theclassification and detection result. The “group of the images” is a setof a plurality of consecutive 2D images within the range indicated bylong rectangles extending in the left-right direction of the human body,numbered [1] to [11] in FIG. 10 (FIG. 10 illustrates only the range ofimages in each group and does not illustrate individual images withineach group), and the number of images in each group is equal to thenumber of input channels in the model.

In the example illustrated in FIG. 10 , by the classification anddetection, for example, the image groups [4] to [6] in dark areas of thedrawing are specified as image groups including the liver, and the otherimage groups [1] to [3] and [7] to [11] are specified as image groupswithout the liver. Among the image groups [4] to [6] including theliver, each of the image group [4] and the image group [6] includes aportion including a sudden change from an image without the liver to animage including the liver and a portion including a sudden change froman image including the liver to an image without the liver. On the otherhand, in the image group [5], each image includes the liver and includesno portion where the sudden change has occurred. Consequently, thespecifying device 20 according to the second embodiment specifies theimage group [4] and the image group [6] as the range of the number ofimportant images (first portion), and specifies the other image groups[1] to [3], image group [5], and image groups [7] to [11] as the rangeof the number of unimportant images (second portion).

FIG. 11 is a schematic diagram illustrating a processing status at stepS100 of the deep learning model of the image processing system accordingto the second embodiment.

As illustrated in FIG. 11 , at step S100, the specifying device 20 mayinfer a classification result of one group of images at a time by themulti-channel deep learning model. For example, when the number ofchannels is set to 10, 10 2D images are used as input to the model, anda result of whether the liver is present in the 10 images is obtainedthrough one-time inference. In the present embodiment, when theinference result of one group of images, for example, the inferenceresult of 10 images, indicates that the liver is present in at least oneimage, it is considered that the liver is present in the group ofimages. By adjusting the number of channels, the number of images ineach group and the corresponding number of groups are adjusted.

Since the inference based on the multi-channel deep learning model hasrelatively high processing speed and low processing load, but has lowclassification accuracy, the inference based on the multi-channel deeplearning model alone may produce an inaccurate result. After performingthe inference based on the multi-channel deep learning model on a seriesof images to be processed, the specifying device 20 according to thesecond embodiment obtains a result indicating whether a sudden change isincluded in a classification result of images of each group in theentire image, specifies, as the range of the number of important images(first portion), an image group including the sudden change in theclassification result, and specifies the other image groups as the rangeof the number of unimportant images (second portion). Subsequently, theimage processing system according to the present embodiment performsmore precise enhanced inference on the range of the number of importantimages (first portion).

With reference to FIG. 12 , the process of step S200 in the secondembodiment is described below.

FIG. 12 is a schematic diagram illustrating a processing status of stepS200 by the image processing system according to the second embodiment.

After the first portion (range of the number of important images) andthe second portion (range of the number of unimportant images) arespecified at step S100, the inference device 30 performs a firstinference process and a second inference process, which are differentinference processes, on the first portion (range of the number ofimportant images) and the second portion (range of the number ofunimportant images), respectively, at step S200. Specifically, asillustrated in FIG. 12 , the inference device 30 performs enhancedinference on the specified first portion as the first inference process.As the enhanced inference, inference based on a single-channel deeplearning model can be performed on each image in the specified firstportion, as illustrated in FIG. 12 . By performing more preciseinference on each image in the first portion by using the single-channeldeep learning model, a more accurate classification result for eachimage can be obtained.

Although FIG. 12 illustrates an example in which the inference based onthe single-channel deep learning model is performed as the enhancedinference; however, the present embodiment is not limited thereto andother enhanced inference methods for classification known to thoseskilled in the art may be employed as the enhanced inference in thesecond embodiment.

The inference device 30 performs light inference, which has a lowerprocessing load and higher speed, on the second portion as the secondinference process. As in the first embodiment, the inference device 30of the second embodiment can use the results of the light inferenceperformed on the second portion when specifying the first portion andthe second portion at step S100. The present embodiment is not limitedthereto, and another light inference may be newly performed on thesecond portion as the second inference process.

After the inference process is terminated, the present embodimentproceeds to the integration process of step S300.

With reference to FIG. 13 , the integration process of step S300 in thesecond embodiment is described below.

FIG. 13 is a schematic diagram illustrating a processing status of stepS300 by the image processing system according to the second embodiment.

As illustrated in FIG. 13 , at step S300, the integration device 40integrates the results of the enhanced inference on the first portionand the light inference on the second portion obtained at step S200, anduses the integrated result as a complete inference result.

Since details of step S300 are the same as in the first embodiment,redundant description is omitted.

As illustrated in FIG. 13 , in accordance with the image processingsystem according to the second embodiment, a more precise classificationresult such as an image in which the liver is not present at a positionindicated by the broken line and an image in which the liver is presentat a position indicated by the solid line can be obtained in the rangeof the number of important images (first portion) (group [4] and group[6]).

Subsequently, the integration device 40 outputs the processing result ofstep S300 as a final result of the inference process.

In accordance with the image processing system 1 according to the secondembodiment, as in the first embodiment, a determination step S400 and acorrection step S500 may be further provided between step S300 and stepS30 as a preferred form as described above.

Details of the process of the determination step S400 are the same asdescribed in the first embodiment with reference to FIG. 3 . Withreference to FIG. 14 , the process of the correction step S500 of thesecond embodiment is described below.

FIG. 14 is a schematic diagram illustrating a processing status of stepS500 by the image processing system according to the second embodiment.

As illustrated in FIG. 14 , the process of step S500 can be implementedby adjusting the boundary of the range of the number of important images(first portion). As in the first embodiment, such adjustment may beautomatically performed by the correction function of the imageprocessing system according to preset rules, or may be implemented by auser who clicks a corresponding region in the image with a mouse anddrags and drops the region.

After step S500 is performed, the image processing system according tothe present embodiment further performs the processes of step S200 andstep S300 on the corrected range of the number of important images(first portion) and the range of the number of unimportant images(second portion) other than the range of the number of important imagesuntil the determination result of step S400 is “YES”, and performs thedetermination of step S400 again on the integrated algorithm result.

Also in the present embodiment, step S400 and step S500 are notessential as in the first embodiment. Even though the processes of stepS400 and step S500 are not performed, the object of the presentembodiment can be similarly achieved and the effects of the presentembodiment can be similarly obtained.

Summary of Second Embodiment

According to the second embodiment, the inference process of the presentembodiment is for positioning an organ included in an image on the basisof image classification. The receiving step receives a plurality ofconsecutive images. The first portion is the range of the number ofimportant images, which is the range of the number of images importantfor the classification of a plurality of images, and the second portionis the range of the number of unimportant images, which is the range ofthe number of images not important for the classification of theplurality of images. The specifying step classifies and detects theplurality of images by using a multi-channel deep learning model, andspecifies, as the range of the number of important images, the range ofa predetermined number of images near an image in which theclassification and detection result suddenly changes among the pluralityof images, and specifies the range of the number of other images as therange of the number of unimportant images. The inference step classifiesand detects the range of the number of important images by using asingle-channel deep learning model, uses a result of the classificationand detection as an inference result for the range of the number ofimportant images, and uses the classification and detection result forthe range of the number of unimportant images at the specifying step asis as the inference result for the range of the number of unimportantimages. In this way, according to the present embodiment, as in thefirst embodiment, an image to be processed is divided into the firstportion and the second portion other than the first portion, and thefirst inference process and the second inference process, which aredifferent inference processes, on the first portion and the secondportion, respectively. Specifically, precise enhanced inference isperformed on the first portion and faster normal inference is performedon the second portion. Subsequently, results of the enhanced inferenceand the normal inference are integrated to obtain a processing resultfor the entire image. This makes it possible to implement a balancebetween accuracy and speed during deep learning model inference, andachieve more desirable inference results.

As described above, the image processing system according to the presentembodiment can implement a balance between accuracy and speed duringdeep learning model inference. For example, in the first embodiment forimage segmentation, the number of image patches for image inference inthe present embodiment can be reduced to ½ or less of that in therelated art without changing the inference accuracy. In the secondembodiment for image classification, the inference accuracy is notchanged, and taking 1000 whole-body CT images as an example, when lightinference is performed once for every 10 images and enhanced inferenceis performed for every one image, the order of the image inferenceprocess in the present embodiment can be reduced to 20% or less of thatin the related art.

OTHER EMBODIMENTS

In the embodiments described above, segmentation and classification ofthe liver have been described as an example; however, the presentembodiment can also be applied to segmentation and classification ofother organs and tissue structures other than the liver, and other typesof image processing other than the segmentation and the classification.

Since the image processing, segmentation, classification, deep learningmodels, neural network training and inference, and the like described inthe embodiments above can all be implemented using various methods inthe related art, detailed description thereof is omitted.

The present embodiment may be implemented as the image processing systemdescribed above, or as an image processing method, an image processingprogram, or a storage medium storing the image processing program.

The image processing system according to the present application may beincorporated into a medical image diagnostic apparatus, or may performprocessing independently. In such a case, the image processing systemincludes processing circuitry that performs the same process as in eachof the steps described above, and a memory storing computer programscorresponding to respective functions, various types of information, andthe like. The processing circuitry acquires 2-dimensional or3-dimensional medical image data from a medical image diagnosticapparatus such as an ultrasonic diagnostic apparatus or an image storageapparatus via a network, and performs the process described above byusing the acquired medical image data. The processing circuitry is aprocessor that reads the computer programs from the memory and executesthe read computer programs, thereby implementing functions correspondingto the executed computer programs.

The term “processor” used in the description of the embodiment describedabove, for example, means circuitry such as a central processing unit(CPU), a graphics processing unit (GPU), an application specificintegrated circuit (ASIC), or a programmable logic device (for example,a simple programmable logic device (SPLD), a complex programmable logicdevice (CPLD), and a field programmable gate array (FPGA)). Instead ofstoring the computer programs in the storage circuitry, the computerprograms may be directly incorporated in the circuitry of the processor.In this case, the processor implements the functions by reading andexecuting the computer programs incorporated in the circuitry. Eachprocessor of the present embodiment is not limited to being configuredas single piece of circuitry for each processor, and one processor maybe configured by combining a plurality of pieces of independentcircuitry to implement the functions thereof.

Each component of each device illustrated in the description of theabove embodiment is functionally conceptual, and does not necessarilyhave to be physically configured as illustrated in the drawings. Thatis, the specific form of distribution or integration of each device isnot limited to that illustrated in the drawings, but can be configuredby functionally or physically distributing or integrating all or partthereof in arbitrary units, depending on various loads, usageconditions, and the like. Moreover, each processing function performedby each device can be implemented in whole or in part by a CPU and acomputer program that is analyzed and executed by the CPU, or byhardware using wired logic.

The processing methods described in the embodiments above can beimplemented by executing a processing program prepared in advance on acomputer such as a personal computer or a workstation. The processingprogram can be distributed via a network such as the Internet. Theprocessing program is recorded on a non-transitory computer readablestorage medium such as a hard disk, a flexible disk (FD), a compact diskread only memory (CD-ROM), a MO, a digital versatile disc (DVD), auniversal serial bus (USB) memory, and a flash memory such as a securedigital (SD) card memory, and can also be executed by being read fromthe non-transitory storage medium by a computer.

Various types of data handled in the present specification are typicallydigital data.

According to at least one embodiment described above, it is possible toimplement a balance between accuracy and speed during deep learningmodel inference, and achieve more desirable inference results.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. An image processing method of performing aninference process on image data by using a trained deep learning model,the image processing method comprising: receiving an image; setting ananatomical site of a target; specifying a first portion and a secondportion in the image, the first portion including a region correspondingto the anatomical site of the target, the second portion including aregion different from the anatomical site of the target; by using thedeep learning model, performing, as a first inference process, asegmentation process of specifying the region corresponding to theanatomical site of the target with respect to the first portion andperforming, as a second inference process, a segmentation process ofspecifying the region different from the anatomical site of the targetwith respect to the second portion, or performing, as the firstinference process, a classification process of classifying and detectingan image including the region corresponding to the anatomical site ofthe target with respect to the first portion and performing, as thesecond inference process, a classification process of classifying anddetecting an image including the region different from the anatomicalsite of the target with respect to the second portion; and integratingresults of the first inference process and the second inference processand outputs an integrated result.
 2. The image processing methodaccording to claim 1, wherein the specifying of the first portion andthe second portion includes performing segmentation on the image byusing the deep learning model, acquiring an inference result indexindicating a segmentation result for each pixel in the image,specifying, as the first portion, a region including pixels for whichthe inference result index is between a predetermined lower limit valueand a predetermined upper limit value, and specifying a region otherthan the first portion as the second portion, and the performing of thefirst inference process and the second inference process includesperforming enhanced segmentation based on enhanced inference on thefirst portion, using a result of the enhanced segmentation as aninference result for the first portion, and using a segmentation resultfor the second portion at the specifying as is as an inference resultfor the second portion.
 3. The image processing method according toclaim 2, wherein the inference result index is an inference probability.4. The image processing method according to claim 2, wherein theinference result index is inference uncertainty.
 5. The image processingmethod according to claim 2, wherein the enhanced inference performs atleast one of dividing image patches used during the inference processaround the first portion, performing the inference again after flippingthe image patch including the first portion, and increasing a patchoverlap ratio between the image patches when performing the inferenceprocess on the first portion.
 6. The image processing method accordingto claim 1, wherein the receiving of the image includes receiving aplurality of consecutive images, the specifying of the first portion andthe second portion includes classifying and detecting the plurality ofimages by using a multi-channel deep learning model, specifying, as thefirst portion, a range of a predetermined number of images near an imagein which a classification and detection result suddenly changes amongthe plurality of images, and specifying a range of the number of otherimages as the second portion, and the performing of the first inferenceprocess and the second inference process includes classifying anddetecting the first portion by using a single channel deep learningmodel, using a result of the classification and detection as aninference result for the first portion, and using a classification anddetection result for the second portion at the specifying as is as aninference result for the second portion.
 7. The image processing methodaccording to claim 1, further comprising correcting the integratedresult by adjusting the first portion.
 8. An image processing systemthat performs an inference process on image data by using a trained deeplearning model, the image processing system comprising: a receivingdevice configured to receive an image; a specifying device configured toset an anatomical site of a target, and to specify a first portion and asecond portion in the image, the first portion including a regioncorresponding to the anatomical site of the target, the second portionincluding a region different from the anatomical site of the target; aninference device configured to, by using the deep learning model,perform, as a first inference process, a segmentation process ofspecifying the region corresponding to the anatomical site of the targetwith respect to the first portion and perform, as a second inferenceprocess, a segmentation process of specifying the region different fromthe anatomical site of the target with respect to the second portion, orto perform, as the first inference process, a classification process ofclassifying and detecting an image including the region corresponding tothe anatomical site of the target with respect to the first portion andperform, as the second inference process, a classification process ofclassifying and detecting an image including the region different fromthe anatomical site of the target with respect to the second portion;and an integration device configured to integrate results of the firstinference process and the second inference process and outputs anintegrated result.
 9. A non-transitory computer readable medium storinga plurality of instructions that is executable by a computer andperforms an inference process on image data by using a trained deeplearning model, wherein the plurality of instructions causes thecomputer to execute: receiving an image; setting an anatomical site of atarget; specifying a first portion and a second portion in the image,the first portion including a region corresponding to the anatomicalsite of the target, the second portion including a region different fromthe anatomical site of the target; by using the deep learning model,performing, as a first inference process, a segmentation process ofspecifying the region corresponding to the anatomical site of the targetwith respect to the first portion and performing, as a second inferenceprocess, a segmentation process of specifying the region different fromthe anatomical site of the target with respect to the second portion, orperforming, as the first inference process, a classification process ofclassifying and detecting an image including the region corresponding tothe anatomical site of the target with respect to the first portion andperforming, as the second inference process, a classification process ofclassifying and detecting an image including the region different fromthe anatomical site of the target with respect to the second portion;and integrating results of the first inference process and the secondinference process and outputs an integrated result.